{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2020 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a target=\"_blank\" href=\"https://colab.research.google.com/github/GoogleCloudPlatform/keras-idiomatic-programmer/blob/master/books/deep-learning-design-patterns/Workshops/Novice/Deep%20Learning%20Design%20Patterns%20-%20Workshop%20-%20Chapter%20II.ipynb\">\n",
    "<img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Learning Design Patterns - Code Labs\n",
    "\n",
    "## Lab Exercise #2 - Get Familiar with Convolutional Neural Networks (CNN)\n",
    "\n",
    "## Prerequistes:\n",
    "\n",
    "    1. Familiar with Python\n",
    "    2. Completed Chapter II: Convolutional Neural Networks\n",
    "\n",
    "## Objectives:\n",
    "\n",
    "    1. Create a basic CNN.\n",
    "    2. Create a VGG class CNN\n",
    "    3. Create a CNN with an identity link (Residual CNN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic CNN as Sequential API\n",
    "\n",
    "Let's create a basic CNN. We will make it as two convolutional layers, each followed by a max pooling layer.\n",
    "\n",
    "We will use these approaches:\n",
    "\n",
    "    1. We will double the number of filters with each subsequent layer.\n",
    "    2. We will reduce the size of the feature maps by using a stride > 1.\n",
    "    \n",
    "You fill in the blanks (replace the ??), make sure it passes the Python interpreter, and then verify it's correctness with the summary output.\n",
    "\n",
    "You will need to:\n",
    "\n",
    "    1. Set the number of channels on the input vector (i.e., input shape).\n",
    "    2. Set the number of filters and stride on the convolutional layers.\n",
    "    3. Set the max pooling window size and stride."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Dense\n",
    "\n",
    "# Let's start with a Sequential model\n",
    "model = Sequential()\n",
    "\n",
    "# Let's assume we are building a model for CIFAR-10, which are 32x32 RGB images\n",
    "# HINT: how many channels are in an RGB image\n",
    "input_shape=(32, 32, ??)\n",
    "\n",
    "# Let's add a first convolution layer with 16 filters of size 3x3 and stride of 2\n",
    "# HINT: first parameter is the number of filters and the second is the filter (kernel) size\n",
    "model.add(Conv2D(??, ??, strides=2, activation='relu', input_shape=input_shape))\n",
    "\n",
    "# Let's reduce the feature maps by 75%\n",
    "# HINT: 2x2 window and move 2 pixels at a time\n",
    "model.add(MaxPooling2D(??, strides=??))\n",
    "\n",
    "# Let's add a second convolution layer with 3x3 filter and strides=2 and double the filters\n",
    "# HINT: double the number of filters you specified in the first Conv2D\n",
    "model.add(Conv2D(??, ??, strides=2, activation='relu'))\n",
    "\n",
    "# Let's reduce the feature maps by 75%\n",
    "model.add(MaxPooling2D(??, strides=??))\n",
    "\n",
    "model.add(Dense(10, activation='softmax'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the model architecture using summary method\n",
    "\n",
    "It should look like below:\n",
    "\n",
    "```\n",
    "Layer (type)                 Output Shape              Param #   \n",
    "=================================================================\n",
    "conv2d_4 (Conv2D)            (None, 15, 15, 16)        448       \n",
    "_________________________________________________________________\n",
    "max_pooling2d_4 (MaxPooling2 (None, 7, 7, 16)          0         \n",
    "_________________________________________________________________\n",
    "conv2d_5 (Conv2D)            (None, 3, 3, 32)          4640      \n",
    "_________________________________________________________________\n",
    "max_pooling2d_5 (MaxPooling2 (None, 1, 1, 32)          0         \n",
    "_________________________________________________________________\n",
    "dense_3 (Dense)              (None, 1, 1, 10)          330       \n",
    "=================================================================\n",
    "Total params: 5,418\n",
    "Trainable params: 5,418\n",
    "Non-trainable params: 0\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## VGG16 as Sequential API\n",
    "\n",
    "Next, we will create a VGG convolutional network. VGG networks are sequential, but they add the concept of convolutional groups. The basic elements of a VGG are:\n",
    "\n",
    "    1. Each convolutional group consists of two or more convolutional layers.\n",
    "    2. Max pooling is deferred to the end of the convolutional group.\n",
    "    3. Each convolutional group is the same or double the number of filters as the last  \n",
    "       group.\n",
    "    4. Multiple dense layers are used for the classifer.\n",
    "    \n",
    "You will need to:\n",
    "\n",
    "    1. Set the number of filters, filter size and padding on the stem convolutional group.\n",
    "    2. Set the the number of filters for the convolutional blocks.\n",
    "    3. Add the flattening layer between the feature learning and classifier groups.\n",
    "    4. Set the number of nodes in the dense layers of the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense\n",
    "\n",
    "def conv_block(n_layers, n_filters):\n",
    "    \"\"\"\n",
    "        n_layers : number of convolutional layers\n",
    "        n_filters: number of filters\n",
    "    \"\"\"\n",
    "    for n in range(n_layers):\n",
    "        model.add(Conv2D(n_filters, (3, 3), strides=(1, 1), padding=\"same\",\n",
    "                  activation=\"relu\"))\n",
    "    model.add(MaxPooling2D(2, strides=2))\n",
    "\n",
    "# Create a Sequential Model\n",
    "model = Sequential()\n",
    "\n",
    "# Add Convolutional Frontend with 64 3x3 filters of stride 1\n",
    "# Set the padding so when the filter is slid over the edges of the image, the \"imaginary\" pixels have the same\n",
    "# value as the pixels on the edge.\n",
    "model.add(Conv2D(??, ??, strides=(1, 1), padding=??, activation=\"relu\",\n",
    "          input_shape=(224, 224, 3)))\n",
    "\n",
    "\n",
    "# These are the convolutional groups - double the number of filters on each progressive group\n",
    "conv_block(1, 64)\n",
    "conv_block(2, ??)\n",
    "conv_block(3, ??)\n",
    "\n",
    "# The last two groups in a VGG16, its double the size of the previous of the group, but both groups are the same size.\n",
    "# HINT: the number should be the same for both\n",
    "conv_block(3, ??)\n",
    "conv_block(3, ??)\n",
    "\n",
    "# Add layer to transistion from final 2D feature maps (bottleneck layer) to 1D vector for DNN.\n",
    "# HINT: think of what you need to do to the 2D feature maps from the convolutional layers before passing to dense layers.\n",
    "model.add(??)\n",
    "# Add DNN Backend with two layers of 4096 nodes\n",
    "# HINT: \n",
    "model.add(Dense(??, activation='relu'))\n",
    "model.add(Dense(??, activation='relu'))\n",
    "\n",
    "# Output layer for classification (1000 classes)\n",
    "model.add(Dense(1000, activation=??))\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the model architecture using summary method\n",
    "\n",
    "It should look like below:\n",
    "\n",
    "```\n",
    "_________________________________________________________________\n",
    "Layer (type)                 Output Shape              Param #   \n",
    "=================================================================\n",
    "conv2d_14 (Conv2D)           (None, 224, 224, 64)      1792      \n",
    "_________________________________________________________________\n",
    "conv2d_15 (Conv2D)           (None, 224, 224, 64)      36928     \n",
    "_________________________________________________________________\n",
    "max_pooling2d_6 (MaxPooling2 (None, 112, 112, 64)      0         \n",
    "_________________________________________________________________\n",
    "conv2d_16 (Conv2D)           (None, 112, 112, 128)     73856     \n",
    "_________________________________________________________________\n",
    "conv2d_17 (Conv2D)           (None, 112, 112, 128)     147584    \n",
    "_________________________________________________________________\n",
    "max_pooling2d_7 (MaxPooling2 (None, 56, 56, 128)       0         \n",
    "_________________________________________________________________\n",
    "conv2d_18 (Conv2D)           (None, 56, 56, 256)       295168    \n",
    "_________________________________________________________________\n",
    "conv2d_19 (Conv2D)           (None, 56, 56, 256)       590080    \n",
    "_________________________________________________________________\n",
    "conv2d_20 (Conv2D)           (None, 56, 56, 256)       590080    \n",
    "_________________________________________________________________\n",
    "max_pooling2d_8 (MaxPooling2 (None, 28, 28, 256)       0         \n",
    "_________________________________________________________________\n",
    "conv2d_21 (Conv2D)           (None, 28, 28, 512)       1180160   \n",
    "_________________________________________________________________\n",
    "conv2d_22 (Conv2D)           (None, 28, 28, 512)       2359808   \n",
    "_________________________________________________________________\n",
    "conv2d_23 (Conv2D)           (None, 28, 28, 512)       2359808   \n",
    "_________________________________________________________________\n",
    "max_pooling2d_9 (MaxPooling2 (None, 14, 14, 512)       0         \n",
    "_________________________________________________________________\n",
    "conv2d_24 (Conv2D)           (None, 14, 14, 512)       2359808   \n",
    "_________________________________________________________________\n",
    "conv2d_25 (Conv2D)           (None, 14, 14, 512)       2359808   \n",
    "_________________________________________________________________\n",
    "conv2d_26 (Conv2D)           (None, 14, 14, 512)       2359808   \n",
    "_________________________________________________________________\n",
    "max_pooling2d_10 (MaxPooling (None, 7, 7, 512)         0         \n",
    "_________________________________________________________________\n",
    "flatten_2 (Flatten)          (None, 25088)             0         \n",
    "_________________________________________________________________\n",
    "dense_4 (Dense)              (None, 4096)              102764544 \n",
    "_________________________________________________________________\n",
    "dense_5 (Dense)              (None, 4096)              16781312  \n",
    "_________________________________________________________________\n",
    "dense_6 (Dense)              (None, 1000)              4097000   \n",
    "=================================================================\n",
    "Total params: 138,357,544\n",
    "Trainable params: 138,357,544\n",
    "Non-trainable params: 0\n",
    "__________________________\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Residual CNN as Functional API\n",
    "\n",
    "Finally, we will create a residual convolutional network (ResNet). The basic elements of a ResNet are:\n",
    "\n",
    "    1. A stem convolutional group of 7x7 filter size.\n",
    "    2. A sequence of residual blocks, where each doubles the number of filters.\n",
    "        A. Each residual block consists of two 3x3 filters, w/o max pooling.\n",
    "        B. The input to the residual block is added to the output.\n",
    "    3. Between residual blocks is a convolutional block that doubles the number of \n",
    "       filters from the previous block, so the number of filters coming in and going \n",
    "       out are the same for the identity link matrix add operation.\n",
    "        A. Each convolutional block consists of two 3x3 filters, but uses stride=2 \n",
    "           to downsample the size of the feature maps.\n",
    "           \n",
    "You will need to:\n",
    "    1. Save the input to the residual block for the identity link.\n",
    "    2. Complete the matrix add of the identity link to the output of the residual block.\n",
    "    3. Set (double) the filters for the convolutional block between residual block groups to match filter sizes for matrix add operations.\n",
    "    4. Add the global averaging layer between the feature learning groups and the classifier.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Model\n",
    "import tensorflow.keras.layers as layers\n",
    "\n",
    "def residual_block(n_filters, x):\n",
    "    \"\"\" Create a Residual Block of Convolutions\n",
    "        n_filters: number of filters\n",
    "        x        : input into the block\n",
    "    \"\"\"\n",
    "    # Save the input as the shortcut for the identity link\n",
    "    # Hint: read the comment on the params to the function.\n",
    "    shortcut = ??\n",
    "    x = layers.Conv2D(n_filters, (3, 3), strides=(1, 1), padding=\"same\",\n",
    "                      activation=\"relu\")(x)\n",
    "    x = layers.Conv2D(n_filters, (3, 3), strides=(1, 1), padding=\"same\",\n",
    "                      activation=\"relu\")(x)\n",
    "    # Add the saved input (identity link) to the output.\n",
    "    # HINT: the name of the variable you used above to save the input.\n",
    "    x = layers.Add()([??, x])\n",
    "    return x\n",
    "\n",
    "def conv_block(n_filters, x):\n",
    "    \"\"\" Create Block of Convolutions without Pooling\n",
    "        n_filters: number of filters\n",
    "        x        : input into the block\n",
    "    \"\"\"\n",
    "    x = layers.Conv2D(n_filters, (3, 3), strides=(2, 2), padding=\"same\",\n",
    "                      activation=\"relu\")(x)\n",
    "    x = layers.Conv2D(n_filters, (3, 3), strides=(2, 2), padding=\"same\",\n",
    "                      activation=\"relu\")(x)\n",
    "    return x\n",
    "\n",
    "# The input tensor\n",
    "inputs = layers.Input(shape=(224, 224, 3))\n",
    "\n",
    "# First Convolutional layer, where pooled feature maps will be reduced by 75%\n",
    "x = layers.Conv2D(64, kernel_size=(7, 7), strides=(2, 2), padding='same', activation='relu')(inputs)\n",
    "x = layers.MaxPool2D(pool_size=(3, 3), strides=(2, 2), padding='same')(x)\n",
    "\n",
    "# First Residual Block Group of 64 filters\n",
    "for _ in range(3):\n",
    "    x = residual_block(64, x)\n",
    "\n",
    "# Double the size of filters and reduce feature maps by 75% (strides=2, 2) to fit the next Residual Group\n",
    "# HINT: number should be twice as big as the number of filters in prior residual_blocks.\n",
    "x = conv_block(??, x)\n",
    "\n",
    "# Second Residual Block Group of 128 filters\n",
    "for _ in range(3):\n",
    "    x = residual_block(128, x)\n",
    "\n",
    "# Double the size of filters and reduce feature maps by 75% (strides=2, 2) to fit the next Residual Group\n",
    "# HINT: number should be twice as big as the number of filters in prior residual_blocks.\n",
    "x = conv_block(??, x)\n",
    "\n",
    "# Third Residual Block Group of 256 filters\n",
    "for _ in range(5):\n",
    "    x = residual_block(256, x)\n",
    "\n",
    "# Double the size of filters and reduce feature maps by 75% (strides=2, 2) to fit the next Residual Group\n",
    "# HINT: number should be twice as big as the number of filters in prior residual_blocks.\n",
    "x = conv_block(??, x)\n",
    "\n",
    "# Fourth Residual Block Group of 512 filters\n",
    "for _ in range(2):\n",
    "    x = residual_block(??, x)\n",
    "\n",
    "# Add a Global Averaging Pooling (inplace of a Flatten) at the end of all the convolutional residual blocks\n",
    "x = layers.??()(x)\n",
    "\n",
    "# Final Dense Outputting Layer for 1000 outputs\n",
    "outputs = layers.Dense(1000, activation='softmax')(x)\n",
    "\n",
    "model = Model(inputs, outputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Verify the model architecture using summary method\n",
    "\n",
    "It should look like below:\n",
    "\n",
    "```\n",
    "__________________________________________________________________________________________________\n",
    "Layer (type)                    Output Shape         Param #     Connected to                     \n",
    "==================================================================================================\n",
    "input_1 (InputLayer)            (None, 224, 224, 3)  0                                            \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_1 (Conv2D)               (None, 112, 112, 64) 9472        input_1[0][0]                    \n",
    "__________________________________________________________________________________________________\n",
    "max_pooling2d_1 (MaxPooling2D)  (None, 56, 56, 64)   0           conv2d_1[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_2 (Conv2D)               (None, 56, 56, 64)   36928       max_pooling2d_1[0][0]            \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_3 (Conv2D)               (None, 56, 56, 64)   36928       conv2d_2[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "add_1 (Add)                     (None, 56, 56, 64)   0           max_pooling2d_1[0][0]            \n",
    "                                                                 conv2d_3[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_4 (Conv2D)               (None, 56, 56, 64)   36928       add_1[0][0]                      \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_5 (Conv2D)               (None, 56, 56, 64)   36928       conv2d_4[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "add_2 (Add)                     (None, 56, 56, 64)   0           add_1[0][0]                      \n",
    "                                                                 conv2d_5[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_6 (Conv2D)               (None, 56, 56, 64)   36928       add_2[0][0]                      \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_7 (Conv2D)               (None, 56, 56, 64)   36928       conv2d_6[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "add_3 (Add)                     (None, 56, 56, 64)   0           add_2[0][0]                      \n",
    "                                                                 conv2d_7[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_8 (Conv2D)               (None, 28, 28, 128)  73856       add_3[0][0]                      \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_9 (Conv2D)               (None, 14, 14, 128)  147584      conv2d_8[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_10 (Conv2D)              (None, 14, 14, 128)  147584      conv2d_9[0][0]                   \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_11 (Conv2D)              (None, 14, 14, 128)  147584      conv2d_10[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "add_4 (Add)                     (None, 14, 14, 128)  0           conv2d_9[0][0]                   \n",
    "                                                                 conv2d_11[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_12 (Conv2D)              (None, 14, 14, 128)  147584      add_4[0][0]                      \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_13 (Conv2D)              (None, 14, 14, 128)  147584      conv2d_12[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "add_5 (Add)                     (None, 14, 14, 128)  0           add_4[0][0]                      \n",
    "                                                                 conv2d_13[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_14 (Conv2D)              (None, 14, 14, 128)  147584      add_5[0][0]                      \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_15 (Conv2D)              (None, 14, 14, 128)  147584      conv2d_14[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "add_6 (Add)                     (None, 14, 14, 128)  0           add_5[0][0]                      \n",
    "                                                                 conv2d_15[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_16 (Conv2D)              (None, 7, 7, 256)    295168      add_6[0][0]                      \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_17 (Conv2D)              (None, 4, 4, 256)    590080      conv2d_16[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_18 (Conv2D)              (None, 4, 4, 256)    590080      conv2d_17[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_19 (Conv2D)              (None, 4, 4, 256)    590080      conv2d_18[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "add_7 (Add)                     (None, 4, 4, 256)    0           conv2d_17[0][0]                  \n",
    "                                                                 conv2d_19[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_20 (Conv2D)              (None, 4, 4, 256)    590080      add_7[0][0]                      \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_21 (Conv2D)              (None, 4, 4, 256)    590080      conv2d_20[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "add_8 (Add)                     (None, 4, 4, 256)    0           add_7[0][0]                      \n",
    "                                                                 conv2d_21[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_22 (Conv2D)              (None, 4, 4, 256)    590080      add_8[0][0]                      \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_23 (Conv2D)              (None, 4, 4, 256)    590080      conv2d_22[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "add_9 (Add)                     (None, 4, 4, 256)    0           add_8[0][0]                      \n",
    "                                                                 conv2d_23[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_24 (Conv2D)              (None, 4, 4, 256)    590080      add_9[0][0]                      \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_25 (Conv2D)              (None, 4, 4, 256)    590080      conv2d_24[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "add_10 (Add)                    (None, 4, 4, 256)    0           add_9[0][0]                      \n",
    "                                                                 conv2d_25[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_26 (Conv2D)              (None, 4, 4, 256)    590080      add_10[0][0]                     \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_27 (Conv2D)              (None, 4, 4, 256)    590080      conv2d_26[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "add_11 (Add)                    (None, 4, 4, 256)    0           add_10[0][0]                     \n",
    "                                                                 conv2d_27[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_28 (Conv2D)              (None, 2, 2, 512)    1180160     add_11[0][0]                     \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_29 (Conv2D)              (None, 1, 1, 512)    2359808     conv2d_28[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_30 (Conv2D)              (None, 1, 1, 512)    2359808     conv2d_29[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_31 (Conv2D)              (None, 1, 1, 512)    2359808     conv2d_30[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "add_12 (Add)                    (None, 1, 1, 512)    0           conv2d_29[0][0]                  \n",
    "                                                                 conv2d_31[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_32 (Conv2D)              (None, 1, 1, 512)    2359808     add_12[0][0]                     \n",
    "__________________________________________________________________________________________________\n",
    "conv2d_33 (Conv2D)              (None, 1, 1, 512)    2359808     conv2d_32[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "add_13 (Add)                    (None, 1, 1, 512)    0           add_12[0][0]                     \n",
    "                                                                 conv2d_33[0][0]                  \n",
    "__________________________________________________________________________________________________\n",
    "global_average_pooling2d_1 (Glo (None, 512)          0           add_13[0][0]                     \n",
    "__________________________________________________________________________________________________\n",
    "dense_1 (Dense)                 (None, 1000)         513000      global_average_pooling2d_1[0][0] \n",
    "==================================================================================================\n",
    "Total params: 21,616,232\n",
    "Trainable params: 21,616,232\n",
    "Non-trainable params: 0\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "Next, we will train two mini-VGGs (6 and 10 layers) on the CIFAR-10 dataset and compare the results. As we have not covered data preprocessing or training, just following the steps.\n",
    "\n",
    "### VGG (6)\n",
    "\n",
    "Let's make a 6 layer VGG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Conv2D, MaxPooling2D, Flatten, Dense\n",
    "def makeVGG6():\n",
    "    def conv_block(n_layers, n_filters):\n",
    "        \"\"\"\n",
    "            n_layers : number of convolutional layers\n",
    "            n_filters: number of filters\n",
    "        \"\"\"\n",
    "        for n in range(n_layers):\n",
    "            model.add(Conv2D(n_filters, (3, 3), strides=(1, 1), padding=\"same\",\n",
    "                      activation=\"relu\"))\n",
    "        model.add(MaxPooling2D(2, strides=2))\n",
    "        \n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(64, (3, 3), strides=(1, 1), padding='same', activation=\"relu\",\n",
    "              input_shape=(32, 32, 3)))\n",
    "\n",
    "    # These are the convolutional groups\n",
    "    conv_block(1, 64)\n",
    "    conv_block(2, 128)\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(4096, activation='relu'))\n",
    "    model.add(Dense(10, activation='softmax'))\n",
    "    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "    return model\n",
    "\n",
    "vgg6 = makeVGG6()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now check the summary(). You should see 34 million parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vgg6.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### VGG(10)\n",
    "\n",
    "Let's now make a 10 layer VGG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def makeVGG10():\n",
    "    def conv_block(n_layers, n_filters):\n",
    "        \"\"\"\n",
    "            n_layers : number of convolutional layers\n",
    "            n_filters: number of filters\n",
    "        \"\"\"\n",
    "        for n in range(n_layers):\n",
    "            model.add(Conv2D(n_filters, (3, 3), strides=(1, 1), padding=\"same\",\n",
    "                      activation=\"relu\"))\n",
    "        model.add(MaxPooling2D(2, strides=2))\n",
    "        \n",
    "    model = Sequential()\n",
    "    model.add(Conv2D(64, (3, 3), strides=(1, 1), padding='same', activation=\"relu\",\n",
    "              input_shape=(32, 32, 3)))\n",
    "\n",
    "    # These are the convolutional groups\n",
    "    conv_block(1, 64)\n",
    "    conv_block(2, 128)\n",
    "    conv_block(3, 256)\n",
    "    model.add(Flatten())\n",
    "    model.add(Dense(4096, activation='relu'))\n",
    "    model.add(Dense(4096, activation='relu'))\n",
    "    model.add(Dense(10, activation='softmax'))\n",
    "    model.compile(loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['acc'])\n",
    "    return model\n",
    "\n",
    "vgg10 = makeVGG10()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now check the summary(). You should see 35 million parameters. Note how there have nearly the same number of parameters, but the 10 layer VGG is deeper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vgg10.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset\n",
    "\n",
    "Let's get the tf.Keras builtin dataset for CIFAR-10. These are 32x32 color images (3 channels) of 10 classes (airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks). We will preprocess the image data (not covered yet)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import cifar10\n",
    "import numpy as np\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()\n",
    "x_train = (x_train / 255.0).astype(np.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results\n",
    "\n",
    "Let's train both the 6 and 10 layer VGG on CIFAR-10 for 3 epochs and compare the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vgg6.fit(x_train, y_train, epochs=3, batch_size=32, validation_split=0.1, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vgg10.fit(x_train, y_train, epochs=3, batch_size=32, validation_split=0.1, verbose=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observation\n",
    "\n",
    "Notice how the shallower VGG (6 layers) increases in accuracy across all three epochs (??), but the deeper VGG (10) does not and in fact it learns nothing (10% is same as random guessing). \n",
    "\n",
    "While this is not a vanishing gradient (we do not see a NaN on the loss), it does show how early CNN architectures when made deeper became less reliable to converge - not covered yet.\n",
    "\n",
    "If we use a larger image size, like 224x224, we can go more layers because we have more pixel data, but eventually we hit the same problem again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End of Lab Exercise"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-2-2-gpu.2-2.m50",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-2-2-gpu.2-2:m50"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
